Computer systems typically employ one or more interconnects to facilitate communication between system components, such as between processors and memory. Interconnects and/or expansion interfaces may also be used to support built-in and add on devices, such as IO (input/output) devices and expansion cards and the like. For many years after the personal computer was introduced, the primary form of interconnect was a parallel bus. Parallel bus structures were used for both internal data transfers and expansion buses, such as ISA (Industry Standard Architecture), MCA (Micro Channel Architecture), EISA (Extended Industry Standard Architecture) and VESA Local Bus. In the early 1990's Intel Corporation introduced the PCI (Peripheral Component Interconnect) computer bus. PCI improved on earlier bus technologies by not only increasing the bus speed, but also introducing automatic configuration and transaction-based data transfers using shared address and data lines.
As time progressed, computer processor clock rates where increasing at a faster pace than parallel bus clock rates. As a result, computer workloads were often limited by interconnect bottlenecks rather than processor speed. Although parallel buses support the transfer of a large amount of data (e.g., 32 or even 64 bits under PCI-X) with each cycle, their clock rates are limited by timing skew considerations, leading to a practical limit to maximum bus speed. To overcome this problem, high-speed serial interconnects were developed. Examples of early serial interconnects include Serial ATA, USB (Universal Serial Bus), FireWire, and RapidIO.
Another standard serial interconnect that is widely used is PCI Express, also called PCIe, which was introduced in 2004 under the PCIe 1.0 standard. PCIe was designed to replace older PCI and PCI-X standards, while providing legacy support. PCIe employs point-to-point serial links rather than a shared parallel bus architecture. Each link supports a point-to-point communication channel between two PCIe ports using one or more lanes, with each lane comprising a bi-directional serial link. The lanes are physically routed using a crossbar switch architecture, which supports communication between multiple devices at the same time. As a result of its inherent advantages, PCIe has replaced PCI as the most prevalent interconnect in today's personal computers. PCIe is an industry standard managed by the PCI-SIG (Special Interest Group).
Recently, Intel introduced the QuickPath Interconnect® (QPI). QPI was initially implemented as a point-to-point processor interconnect replacing the Front Side Bus on platforms using high-performance processors, such as Intel® Xeon®, and Itanium® processors. QPI is scalable, and is particularly advantageous in systems having multiple processor cores employing shared memory resources.
QPI transactions are facilitated via packetized messages transported over a multi-layer protocol. As shown in FIG. 1, the layers include a Physical layer, a Link layer, a Transport layer, and a Protocol layer. At the Physical layer, data is exchanged in 20-bit phits (Physical Units). At the link layer phits are aggregated into 80-bit flits (flow control units). At the Protocol layer, messages are transferred between agents using a packet-based transport.
The Physical layer defines the physical structure of the interconnect and is responsible for dealing with details of operation of the signals on a particular link between two agents. This layer manages data transfer on the signal wires, including electrical levels, timing aspects, and logical issues involved in sending and receiving each bit of information across the parallel lanes. As shown by QPI architecture 200 in FIG. 2, the physical connectivity of each interconnect link is made up of twenty differential signal pairs plus a differential forwarded clock. Each port supports a link pair consisting of two uni-directional links to complete the connection between two components. This supports traffic in both directions simultaneously.
Components with QPI ports communicate using a pair of uni-directional point-to-point links, defined as a link pair, as shown in FIG. 2. Each port comprises a Transmit (TX) link interface and a Receive (RX) link interface. For the illustrated example, Component A has a TX port that is connected to Component B RX port. One uni-directional link transmits from Component A to Component B, and the other link transmits from Component B to Component A. The “transmit” link and “receive” link is defined with respect to a specific QPI agent. The Component A transmit link transmits data from Component A TX port to Component B RX port. This same Component A transmit link is the Port B receive link.
The second layer up the protocol stack is the Link layer, which is responsible for reliable data transmission and flow control. The Link layer also provides virtualization of the physical channel into multiple virtual channels and message classes. After the Physical layer initialization and training is completed, its logical sub-block works under the direction of the link layer, which is responsible for flow control. From this link operational point onwards, the logical sub-block communicates with the Link layer at a flit granularity (80 bits) and transfers flits across the link at a phit granularity (20 bits). A flit is composed of integral number of phits, where a phit is defined as the number of bits transmitted in one unit interval (UI). For instance, a full-width QPI link transmits and receives a complete flit using four phits. Each flit includes 72 bits of payload and 8 bits of CRC.
QPI employs both a common reference clock at each end of the link, as depicted by a common reference clock 202 in FIG. 2, and forwarded clock signals. The forwarded clock signals are used for phit timing aspects used for data recovery at the physical layer, as depicted in FIG. 3, which shows the relationship between two link layers and their corresponding physical layers. The link layer creates flits to be transmitted, and the physical layer breaks them up into multiple phits using timing provided the forwarded clock signals. These phits are then sent one after the other to the receiver end of the link. The receiver captures this data through use of the forwarded clock signals, and assembles the information back into the flits to pass on to the receiving link layer.
In addition to use for timing purposes, the physical lanes used for the forwarded clock signals are also used to facilitate sideband signaling for various purposes. For example, sometimes the flow of flits are interrupted in order to send control information relevant to the physical condition of the interconnect. To facilitate such an interruption, corresponding signals are sent over the forwarded clock lanes.
While QPI employs forwarded clock signaling, the use of such timing signals is not required for facilitating high speed serial interconnect links. For example, high speed serial interconnects such as PCIe employ 8b/10b encoding, which maps 8-bit symbols to 10-bit symbols to achieve DC balance while providing adequate state changes to facilitate clock recovery.
In view of this, the use of separate forwarded clock signals is optional with respect to link timing operations (acknowledging there are some advantages in using this approach, depending on the physical structure of the interconnect and operational environment). However, there is still a need for sending control information between link ports, which is facilitated, in part, through the use of the forwarded clock signal lanes under QPI, as discussed above.